Add ExternalAudioSource for publishing app audio as an independent track - #1084
Draft
hiroshihorie wants to merge 6 commits into
Draft
Add ExternalAudioSource for publishing app audio as an independent track#1084hiroshihorie wants to merge 6 commits into
hiroshihorie wants to merge 6 commits into
Conversation
Wraps the new RTCExternalAudioSource from the webrtc-sdk fork: an audio source the app pushes AVAudioPCMBuffer/CMSampleBuffer audio into, bypassing the AudioDeviceModule and the mic mixer entirely. LocalAudioTrack.createTrack(externalSource:) publishes it with the screenShareAudio source, and buffers are converted to the declared format via a cached AudioConverter. ScreenShareCaptureOptions gains appAudioPublishMode: .mix keeps the current behavior of summing app audio into the microphone track, .separateTrack routes captured app audio (ReplayKit broadcast on iOS, ScreenCaptureKit on macOS) to an independent track that follows the screen-share video lifecycle, including unpublish on out-of-band stops. Requires a webrtc-xcframework build containing LKRTCExternalAudioSource; the version pin bump lands separately once that release exists.
A missed cleanup or double publish must not leave stale screen-share audio tracks behind on the next stop. Also route the stop path through unpublishAppAudioTrackIfNeeded instead of duplicating the lookup.
On iOS with useBroadcastExtension false the capturer is the in-app capturer, which has no audio handling. The cast to BroadcastScreenCapturer silently returned nil and the separate track was published anyway, permanently silent. Bail with a warning instead, on both platforms. Also tag the E2E test .e2e to match sibling suites and assert the ADM recording path never started, which is the guarantee the test exists to prove.
webrtc-sdk PR 275 added isVoiceProcessingEnabled to the delegate method, so frameworks built past 144.7559.11 fail protocol conformance without this. The parameter is ignored for now, exposing it through AudioEngineObserver belongs to the framework version bump.
With appAudioPublishMode == .separateTrack, a missing appAudioSink (the audio track failed to publish or was unpublished mid-share) fell back to AudioManager.shared.mixer, silently mixing app audio into the microphone track. And when the mic is not published the mixer has no converter, spamming a warning per buffer. Drop the buffers with a single warning instead, in both capturers. Also log an app-audio publish failure and let the screen share continue instead of unwinding after the video track already published, which is exactly the state that leaves the weak sink dangling.
The wait added in #690 observes the ADM capture path via an audio renderer, which external sources bypass by design, so publishing an external track always timed out after the rebase picked the gate up. Frames also only flow once the app pushes audio, so there is nothing to deterministically wait for. The server had already accepted the track, the timeout then tore it down and the capturer's weak sink died, which is how this surfaced.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
ExternalAudioSource, a pushable audio source the app feeds PCM into, so app or screen-share audio can be published as an independent track instead of being mixed into the microphone track inside the engine.Built on webrtc-sdk/webrtc#282, which adds the native
RTCExternalAudioSourceand keeps ADM audio out of externally fed send streams. Draft until a LiveKitWebRTC release containing that lands and the pins here can be bumped. CI is expected to fail against144.7559.11, which lacks the new API and the updated ADM delegate signature.API
ExternalAudioSource: pushAVAudioPCMBufferorCMSampleBuffer, foreign formats are converted via a cachedAudioConverter. Buffered pacing with overflow rejection,bufferedDurationMs,clearBuffer().LocalAudioTrack.createTrack(externalSource:): publishes as.screenShareAudio, andstartCapture()short-circuits so the ADM recording path never starts.ScreenShareCaptureOptions.appAudioPublishMode:.mix(default, existing behavior) or.separateTrack. With.separateTrack, screen share publishes app audio as its own track and unpublishing the video takes the audio track with it.BroadcastScreenCapturer(iOS) andMacOSScreenCapturerfeed the source directly. In.separateTrackmode a missing sink drops audio with a single warning instead of silently falling back into the mic mix. iOS in-app capture has no audio path, so track prep bails with a warning instead of publishing a permanently silent track.Commit guide
0babffe6the feature itself (wrapper, track creation, options, capturer routing, lifecycle, E2E test)d794daa6unpublish all.screenShareAudiopublications, not just the first match6518a07bfail track prep when the capturer has no audio path, tag the E2E test.e2e, assert the ADM recording path never started1bad4614never route separate-track app audio into the mic mixer, log app-audio publish failures and let the share continue8795a447skip the publish frame-wait gate from Ensure audio frame when publishing #690 for externally fed tracks. The watcher observes the ADM capture path, which external sources bypass by design, so the publish always timed outbc739c86adapt to the new ADMwillEnableEnginedelegate signature (webrtc-sdk#275). Required by any framework built past144.7559.11, rides with the pin bumpTesting
ExternalAudioSourceTests4/4 againstlivekit-server --dev, including publish to a remote participant with the ADM recording path asserted never startedset(source:)publishes video plus an independentscreen_share_audiotrack, remote side gets separate volume control per track, stop unpublishes bothensureDeviceAccessfirst. Needs a decision whether the SDK requests automatically in the publish pathMerge gates
Package.swift,Package@swift-6.2.swift,LiveKitClient.podspec, andPackage.resolvedto that version in this PR